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Playing games my whole life, 
now I get to help people make them 

Formerly head of analytics at Kabam 

° Games worked on: 

° Fast and Furious: Legacy 

° Marvel Contest of Champions 

° Transformers Forged to Fight 

Joined Roblox 2 years ago (2017) 

° Started in Analytics, 
now the head of Growth 


AGENDA 

1. Growth as a Process 

a. How to think about Growing your games (and using analytics to do it) 

2. Growing First-Time Players 

a. Example from Roblox 

b. Example from the Community: Dungeon Quest 



GROWTH AS A PROCESS 


“Growth” is applied analytics for improving the things you build 

It’s the scientific method, but for software 


What are the steps in the scientific method? 
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DIVE DEEPER: THE 4 STEPS 


1. Identify a problem 

a. What does the data tell us we need to focus on improving? 

2. Form a hypothesis 

a. What are all of the different things we could change to help? 

3. Perform an Experiment 

a. Try different solutions to the problem 


4. Analyze the Results 

a. Get results back and determine if it worked, and repeat the process 





Growing First Time Players 


EXAMPLE FROM ROBLOX 


HOW CAN WE GROW ROBLOX? 


■ Roblox has similar goals to our developers: grow the total number 
of people playing 

° The more people we get to sign up for Roblox, the more people we have 
to explore the worlds you’ve built 


One way to grow Roblox is by increasing the number of people who make 
it through the signup process 




STEP 1: IDENTIFY THE PROBLEM 


■ Step 1: Identify the Problem 

° How do we determine where we can improve our sign up process? 

° We use analytics to create a view of our user’s called a “Funnel” 


■ The current sign up flow for users on Android: 


























































IDENTIFY CONTINUED: TAGGING YOUR USERS 


■ We need to tag our users based on if they complete any of the steps 

° We’ll need to create events to fire into a datastore 

■ Playfab code is shown below 

° Can also use datastores (with some restrictions) or any other 3rd party database 


1 AnalyticsService game:GetService( M AnalyticsService") 

2 

3 — Event parameters 

4 eventCategory "stepCompleted" 

5 eventValue { 

6 StepName "01" 

7 } 

8 — Fire the event 

9 AnalyticsService:FireEventjeventCategory, eventValuejJ 


■ Full sample code for tagging and testing your users is published in the 
appendix for the talk as a module on Roblox 



IDENTIFY CONTINUED: FUNNEL VIEWS 


Once we’ve tagged our users, we create a table that 
shows which steps the user’s made it through 





Step 1: Step 2: Step 3: Step 4: Click Step 5: 

Birthday Username Password Sign Up Button Gender 


■ We can do this with SQL or just manually 
typing the totals into a spreadsheet 


Then we graph the totals to see where 
users are dropping off 





# of users 


IDENTIFY CONTINUED: FUNNEL VIEWS 


■ Clearly we had a big drop off at the Username step 



Step 1: Birthday Step 2: Username Step 3: Password Step 4: Click Sign Up Step 5: 

Button Gender 






STEP 2: FORM A HYPOTHESIS 


■ Why do we think people are dropping off? 

" Inappropriate names not going through? 

" Unsure about what to do? 

" Can’t find a username 
that isn’t taken? 


■ Hypotheses for improving the drop off at the 
“Username” step 

” Username “clean up” 

° “Display” names 

- Suggested Username feature 
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# of users 


Stepl: Step 2: Step 3: Step 4: Click Sign Step 5: 

Birthday Username Password Up Button Gender 






STEP 3: PERFORM AN EXPERIMENT 

■ How do you know if what you built fixes the problem you identified? 

■ 2 Types of experiments 

° A/B Testing 

° Before / After (look at your funnel again) 




STEP 3: EXPERIMENTS WITH A/B TESTING 

WHAT IS AN A/B TEST? 

■ Randomly give a certain percentage of your users the new experience 
you’ve built (test), and leave the rest with the original (control) experience 

■ Example: give 50% of new user’s the suggested username feature , 
and 50% keep the original sign up flow 

WHAT IS BEFORE / AFTER TESTING? 

■ Visualize the funnel separately based on when the users went through it 

■ Control is before you shipped the new change, and Test is after 





STEP 3: EXPERIMENTS WITH A/B TESTING 


HOW DO YOU KNOW WHICH A/B VERSION IS BEST? 

■ It’s always best to test the differences between the groups for statistical significance 

° This is outside the scope of this presentation 

■ Visualize the funnel separately based on which version a user got 


Variant A 
(Control) 


Variant B 
(Test) 



STEP 3: EXPERIMENTS WITH BEFORE / AFTER TESTING 

■ What if I can’t figure out how to do A/B testing? 

° You can look at the funnel before & after the change 

■ Watch out: easy to be fooled by seasonality and random chance 
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STEP 4: ANALYZE & REPEAT 

■ Looks like the results worked 

■ Analysis: reducing friction is important 

° User’s have a clear intent (play Roblox) so any way to reduce 
friction between them and doing that action could be positive 

■ Repeat. There are other places in the app with friction 


For instance: load times, or monetization funnels 

If we work to remove friction, we’ll get more people 
to play and we’ll continue to grow 





GROWING FIRST TIME PLAYERS: 

Example from the Community: Dungeon Quest 
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BACKGROUND 


■ Dungeon Quest is an RPG on Roblox 
- The core loop is similar to many dungeon craw 



LEVEL UP 


RPGs 


N A DUNGEON 

f 


KILL MONSTERS 


NOW LET’S TRY THIS AS A DEVELOPER ON ROBLOX* 


■ How could we grow our game? 

° As before, we want to focus on improving something that affects all users 

° Instead of focusing on getting more users through a signup process, 

we can improve our first-time user experience with funnel analysis 
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k AII of this data is simuiated and we’ve picked them because we really love their game 
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STEP 1: IDENTIFY THE PROBLEM 

■ What are the steps that we think all users should complete the first time they play 
Dungeon Quest? This is what it looks like when you load into the game at a low level 












IDENTIFY CONTINUED: TAGGING YOUR USERS 

■ We need to tag our users based on if they complete any of the steps 

° A game like Dungeon Quest is so popular they would likely need to use PlayFab or 
another external database 

■ PlayFab code is shown below again (this is pretty easy!) 


1 AnalyticsService game:GetService( M AnalyticsService") 

2 

3 — Event parameters 

4 eventCategory "stepCompleted" 

5 eventValue { 

6 StepName = "01" 

7 } 

8 — Fire the event 

9 AnalyticsService:FireEvent^eventCategory, eventValuejJ 


■ Full sample code for tagging and testing your users is published in the appendix 
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IDENTIFY CONTINUED: FUNNEL VIEW 


■ Create the table like before, and visualize the drop offs 


UserlD 

Press Play >in / Start Dunge* 

Press Start 

Kill Enemy 

Level Up 

348185 

1 1 

1 

1 

1 

216081 

1 0 

0 

0 

0 

168834 

1 1 

1 

1 

1 

127324 

1 1 

1 

1 

1 

401818 

1 1 

1 

1 

1 

246258 

1 0 

0 

0 

0 

126269 

1 1 

1 

1 

1 

100 

95 66.5 

63.5 

60.5 

57.5 



Press Join / Press Kill an Level Up 

Play Start Start Enemy 

Dungeon 


■ As before, we can use either premade tools 
(like Playfab), SQL, or manual counting to 
summarize the data 


Then we graph the totals to see where 
users are dropping off 








FORM A HYPOTHESIS 


■ Why do we think people aren’t making it 
into the Dungeon? 

° Not sure that it’s required 
° Confused how to find them 

° Confused about why they can’t 
join existing dungeons 

■ Options for getting more people into 
the Dungeon 


Press Join / Press Kill an Level Up 

Play Start Start Enemy 

Dungeon 


rR> 


Put up text telling users what to do 
Make the button to Play bigger 

Hide dungeons that users cannot play 




# of users 


STEP 3: EXPERIMENTS WITH A/B TESTING 

HOW DO YOU KNOW WHICH VERSION IS BEST? 

■ Visualize the funnel separately based on which version a user got 


Control Test (New) 




Press 

Join / 

Press 

Kill an 

Level Up 

Press 

Join / 

Press 

Kill an 


Play 

Start 

Start 

Enemy 


Play 

Start 

Start 

Enemy 
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Dungeon 





Dungeon 




STEP 4: ANALYZE & REPEAT 


Looks like the results worked 

Analysis: Extra information can be hard for users to understand 

° People don’t like to feel confused about what they should be doing 

° Always want to show the least amount of information possible without losing the 
message 

Repeat: There are other places in the game with drop off 

° We could look at a similar distribution for how many people 

make it to each level and spot where we’ve made the game too difficult 

° We could look at other elements of the core loop 

° How many people acquire new gear? 





WRAP UP 


After you’ve built a game, if you want to optimize it you can apply the 
Growth Process 

With tools like PlayFab, tagging and visualizing what your users are doing 
can be straightforward 

Funnels are a key visualization tool for diagnosing problems with your work 

With the right testing tools, and enough users, you can (and should) test 
every new feature you build! 




Questions? 







Module with code for A/B testing + Tagging Users 


https://www.roblox.com/librarv/3571275444/AB-Test-Framework 
Credit: Joshua Martheze (AlgyLacey) 



